Udacity - Self-Driving Car NanoDegree

Dean Webb - Vehicle Detection & Tracking Pipeline

In this project, your goal is to write a software pipeline to detect vehicles in a video (start with the test_video.mp4 and later implement on full project_video.mp4), but the main output or product we want you to create is a detailed writeup of the project. Check out the writeup template for this project and use it as a starting point for creating your own writeup.

Project Goals:

The goals / steps of this project are the following:

  • Perform a Histogram of Oriented Gradients (HOG) feature extraction on a labeled training set of images and train a classifier Linear SVM classifier
  • Optionally, you can also apply a color transform and append binned color features, as well as histograms of color, to your HOG feature vector.
  • Note: for those first two steps don't forget to normalize your features and randomize a selection for training and testing.
  • Implement a sliding-window technique and use your trained classifier to search for vehicles in images.
  • Run your pipeline on a video stream (start with the test_video.mp4 and later implement on full project_video.mp4) and create a heat map of recurring detections frame by frame to reject outliers and follow detected vehicles.
  • Estimate a bounding box for vehicles detected.

Here are links to the labeled data for vehicle and non-vehicle examples to train your classifier. These example images come from a combination of the GTI vehicle image database, the KITTI vision benchmark suite, and examples extracted from the project video itself. You are welcome and encouraged to take advantage of the recently released Udacity labeled dataset to augment your training data.

Some example images for testing your pipeline on single frames are located in the test_images folder. To help the reviewer examine your work, please save examples of the output from each stage of your pipeline in the folder called ouput_images, and include them in your writeup for the project by describing what each image shows. The video called project_video.mp4 is the video your pipeline should work well on.

Dataset - Load Data and Extract Features

In [1]:
%matplotlib inline
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import cv2
import glob
from PIL import Image
import time
import os
import pickle
from skimage.feature import hog
from sklearn.svm import LinearSVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
In [2]:
## Dataset Parameters ##
LABELS_CSV = 'FILL_LATER.csv'
TRAINING_DATASET_DIRECTORY = 'training_set/'
PIPELINE_SETUP_DIRECTORY = 'pipeline_setup_images/'
WORKING_DIRECTORY = 'data/'
NON_VEHICLES_TOKEN = 'non-vehicles'
dataset_path = "{}{}{}{}".format(WORKING_DIRECTORY, TRAINING_DATASET_DIRECTORY,'**/', '*.png')
DATACACHE_DIRECTORY = os.path.join(WORKING_DIRECTORY, 'datacache/')

## Image Processing ##
DEFAULT_LENGTH, DEFAULT_WIDTH, DEFAULT_DEPTH = (64, 64, 3)
if DEFAULT_DEPTH > 1:
    DEFAULT_RESOLUTION = (DEFAULT_LENGTH, DEFAULT_WIDTH, DEFAULT_DEPTH)
else:
    DEFAULT_RESOLUTION = (DEFAULT_LENGTH, DEFAULT_WIDTH)

## Feature Extraction Parameters ##
# Spatial Binning
SPATIAL = 32
BIN_SPATIAL_SIZE = (SPATIAL, SPATIAL)
# Color Histogram
HIST_NBINS = 128
COLOR_SPACE = 'HSV'
# HOG Parameters
HOG_ORIENTATIONS = 9
HOG_PIXELS_PER_CELL = 8
HOG_CELLS_PER_BLOCK = 2
HOG_CHANNEL = 'ALL' # Can be 0, 1, 2, or "ALL"

# SVC Parameters
VALIDATION_PORTION = .3
N_PREDICTIONS = 100
In [3]:
# Define a function to scale .PNG and JPEG Files both to 0 to 1 
def normalize_pixels(img):
    max_pixel_value = np.max(img)
    if max_pixel_value > 1.0:
        img = np.copy(np.multiply(img, 1.0 / 255.0)).astype(np.float64) 
    return img

# Define a function to scale .PNG and JPEG Files both to 0 to 1 
def denormalize_pixels(img):
    max_pixel_value = np.max(img)
    if max_pixel_value <= 1.0:
        img = np.copy(np.multiply(img, 255.0)).astype(np.float64) 
    return img
In [4]:
# Define a function to compute binned color features  
def bin_spatial(img, size=BIN_SPATIAL_SIZE):
    color1 = cv2.resize(img[:,:,0], size).ravel()
    color2 = cv2.resize(img[:,:,1], size).ravel()
    color3 = cv2.resize(img[:,:,2], size).ravel()
    return np.hstack((color1, color2, color3)) 
In [5]:
# Define a function to compute color histogram features  
def color_hist(img, nbins=HIST_NBINS):
    # Compute the histogram of the color channels separately
    channel1_hist = np.histogram(img[:,:,0], bins=nbins)
    channel2_hist = np.histogram(img[:,:,1], bins=nbins)
    channel3_hist = np.histogram(img[:,:,2], bins=nbins)
    # Concatenate the histograms into a single feature vector
    hist_features = np.concatenate((channel1_hist[0], channel2_hist[0], channel3_hist[0]))
    return hist_features
In [6]:
# Define a function to return HOG features and visualization --
def get_hog_features(img, orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL,
                     cell_per_block=HOG_CELLS_PER_BLOCK, vis=False, feature_vec=True):      
    denormalized_img = denormalize_pixels(img)
    if vis == True:
        features, hog_image = hog(denormalized_img, orientations=orient, pixels_per_cell=(pix_per_cell, pix_per_cell),
                                  cells_per_block=(cell_per_block, cell_per_block), transform_sqrt=False, 
                                  visualise=vis, feature_vector=feature_vec)
        return features, hog_image
    else:      
        features = hog(denormalized_img, orientations=orient, pixels_per_cell=(pix_per_cell, pix_per_cell),
                       cells_per_block=(cell_per_block, cell_per_block), transform_sqrt=False, 
                       visualise=vis, feature_vector=feature_vec)
        return features
In [7]:
# Define a function to extract features from a list of images
def extract_features(imgs, cspace=COLOR_SPACE, spatial_size=BIN_SPATIAL_SIZE,
                        hist_bins=HIST_NBINS):
    # Create a list to append feature vectors
    features = []
    for file in imgs:
        image = mpimg.imread(file)

        # Image read in from mpimg + .png -> (0 to 1) scaled
        if cspace != 'RGB':
            if cspace == 'HSV':
                feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
            elif cspace == 'LUV':
                feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2LUV)
            elif cspace == 'HLS':
                feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
            elif cspace == 'YUV':
                feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2YUV)
        else: feature_image = np.copy(image)
        
        # Apply bin_spatial() to get spatial color features
        spatial_features = bin_spatial(feature_image, size=spatial_size)
        # Apply color_hist() also with a color space option now
        hist_features = color_hist(feature_image, nbins=hist_bins)
        
        # Call get_hog_features() with vis=False, feature_vec=True
        hog_image = np.copy(image)
        hog_shape = np.asarray(hog_image.shape)
        if HOG_CHANNEL == 'ALL':
            hog_features = []
            for channel in range(len(hog_shape)):
                hog_features.append(get_hog_features(hog_image[:,:,channel]))
            hog_features = np.ravel(hog_features)        
        else:
            hog_features = get_hog_features(hog_image[:,:,HOG_CHANNEL])
        
        # Append the new feature vector to the features list
        features.append(np.concatenate((spatial_features, hist_features, hog_features)))
    # Return list of feature vectors
    return features
In [8]:
## Starting Training Pipeline ##
# Load Image Paths 
images = glob.glob(dataset_path, recursive=True)
cars = []
notcars = []
for image in images:
    if NON_VEHICLES_TOKEN in image:
        notcars.append(image)
    else:
        cars.append(image)
print('Number of Vehicle Images Found:',len(cars),'(images in directory):', dataset_path)
print('Number of Non-Vehicle Images Found:',len(notcars),'(images in directory):', dataset_path)
assert len(images) == len(cars) + len(notcars), 'The subarrays have not split the dataset correctly.'
Number of Vehicle Images Found: 8792 (images in directory): data/training_set/**/*.png
Number of Non-Vehicle Images Found: 8968 (images in directory): data/training_set/**/*.png
In [9]:
# Start Pipeline - Combine and Normalilze Features
car_features = extract_features(cars)
notcar_features = extract_features(notcars)

# Create an array stack of feature vectors
X = np.vstack((car_features, notcar_features)).astype(np.float64)  

# Fit a per-column scaler
X_scaler = StandardScaler().fit(X)

# Apply the scaler to X
scaled_X = X_scaler.transform(X)
car_ind = np.random.randint(0, len(cars))
# Plot an example of raw and scaled features
fig = plt.figure(figsize=(12,4))
plt.subplot(131)
plt.imshow(mpimg.imread(cars[car_ind]))
plt.title('Original Image')
plt.subplot(132)
plt.plot(X[car_ind])
plt.title('Raw Features')
plt.subplot(133)
plt.plot(scaled_X[car_ind])
plt.title('Normalized Features')
fig.tight_layout()

print('Feature Vector size for Cars:', len(car_features[car_ind]))
print('Using Spatial Binning of:',BIN_SPATIAL_SIZE[0],
    'and', HIST_NBINS,'histogram bins')
Feature Vector size for Cars: 8748
Using Spatial Binning of: 32 and 128 histogram bins
In [10]:
# Define the labels vector
y = np.hstack((np.ones(len(car_features)), np.zeros(len(notcar_features))))
In [11]:
# Split up data into randomized training and test sets
rand_state = np.random.randint(0, 100)
X_train, X_test, y_train, y_test = train_test_split(
    scaled_X, y, test_size=VALIDATION_PORTION, random_state=rand_state)
print('Feature vector length:', len(X_train[0]))
Feature vector length: 8748
In [12]:
# Use a linear SVC 
svc = LinearSVC()
# Check the training time for the SVC
t=time.time()
svc.fit(X_train, y_train)
t2 = time.time()
print(round(t2-t, 2), 'Seconds to train SVC...')
# Check the score of the SVC
print('Test Accuracy of SVC = ', round(svc.score(X_test, y_test), 4))
# Check the prediction time for a single sample
t=time.time()
n_predict = N_PREDICTIONS
print('SVC predicts: ', svc.predict(X_test[0:n_predict]))
print('For these',n_predict, 'labels: ', y_test[0:n_predict])
t2 = time.time()
print(round(t2-t, 5), 'Seconds to predict', n_predict,'labels with SVC')
105.35 Seconds to train SVC...
Test Accuracy of SVC =  0.9908
My SVC predicts:  [ 1.  0.  0.  1.  1.  1.  1.  0.  0.  0.  0.  0.  0.  1.  0.  1.  1.  1.
  0.  0.  1.  0.  1.  0.  1.  0.  1.  0.  0.  0.  1.  0.  1.  0.  0.  1.
  1.  0.  1.  1.  0.  1.  1.  1.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.
  1.  1.  0.  0.  1.  1.  0.  1.  0.  1.  1.  0.  0.  1.  0.  1.  1.  0.
  0.  0.  1.  0.  1.  0.  0.  1.  1.  0.  0.  0.  0.  1.  1.  1.  0.  0.
  0.  1.  0.  1.  0.  1.  0.  0.  1.  1.]
For these 100 labels:  [ 0.  0.  0.  1.  1.  1.  1.  0.  0.  0.  0.  0.  0.  1.  0.  1.  1.  1.
  0.  0.  1.  0.  1.  0.  1.  0.  1.  0.  0.  0.  1.  0.  1.  0.  0.  1.
  1.  0.  1.  1.  0.  1.  1.  1.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.
  1.  1.  0.  0.  1.  1.  0.  1.  0.  1.  1.  0.  0.  1.  0.  1.  1.  0.
  0.  0.  1.  0.  1.  0.  0.  1.  1.  0.  0.  0.  0.  1.  1.  1.  0.  0.
  0.  1.  0.  1.  0.  1.  0.  0.  1.  1.]
0.30183 Seconds to predict 100 labels with SVC
In [13]:
#Save Support Vector Classifier to Datacache
def save_to_datacache(support_vector_classifier, datacache_dir=DATACACHE_DIRECTORY):
    os.makedirs(datacache_dir, exist_ok=True)
    svc_pickle = os.path.join(datacache_dir,"svc_pickle.p")
    if not os.path.exists(svc_pickle): 
        svc_hyperparameters = {'svc': svc,
                               'X_scaler':X_scaler,
                               'SPATIAL': SPATIAL, 
                               'HIST_NBINS': HIST_NBINS,
                               'COLOR_SPACE': COLOR_SPACE,
                               'HOG_ORIENTATIONS': HOG_ORIENTATIONS,
                               'HOG_PIXELS_PER_CELL': HOG_PIXELS_PER_CELL,
                               'HOG_CELLS_PER_BLOCK': HOG_CELLS_PER_BLOCK,
                               'HOG_CHANNEL': HOG_CHANNEL
                              }

        pickle.dump(svc_hyperparameters, open(svc_pickle, "wb"))
    
# Save classifier and parameters to datacache directory  
save_to_datacache(svc)

Finished Training SVC - Now to Utilize Sliding Windows

In [14]:
import matplotlib.image as mpimg
import numpy as np
import cv2
from skimage.feature import hog
In [64]:
# Sliding Window Constants
WORKING_DIRECTORY = 'data/'
DATACACHE_DIRECTORY = os.path.join(WORKING_DIRECTORY, 'datacache/')
svc_pickle = os.path.join(DATACACHE_DIRECTORY,"svc_pickle.p")
TESTING_DATASET_DIRECTORY = 'testing_dataset/'
TESTING_PIPELINE_SETUP_DIR= 'test_images/'

testset_path = "{}{}{}".format(WORKING_DIRECTORY, TESTING_PIPELINE_SETUP_DIR, '*.jpg')


with open(svc_pickle, mode='rb') as f:
    svc_hyperparameters = pickle.load(f)
    
    
## Feature Extraction Parameters ##
# Spatial Binning
SVC = svc_hyperparameters['svc']
X_SCALER = svc_hyperparameters['X_scaler']
SPATIAL = svc_hyperparameters['SPATIAL']

BIN_SPATIAL_SIZE = (SPATIAL, SPATIAL)
# Color Histogram
HIST_NBINS = svc_hyperparameters['HIST_NBINS']
COLOR_SPACE = svc_hyperparameters['COLOR_SPACE']
# HOG Parameters
HOG_ORIENTATIONS = svc_hyperparameters['HOG_ORIENTATIONS']
HOG_PIXELS_PER_CELL = svc_hyperparameters['HOG_PIXELS_PER_CELL']
HOG_CELLS_PER_BLOCK = svc_hyperparameters['HOG_CELLS_PER_BLOCK']
HOG_CHANNEL = svc_hyperparameters['HOG_CHANNEL'] # Can be 0, 1, 2, or "ALL"

BBOX_COLOR = (0, 0, 255)
BBOX_THICK = 6

## Sliding Windows Parameters - Horizon ##
SW_YSTART_HORIZON = 368
SW_YSTOP_HORIZON = 512
SW_XY_WINDOW_HORIZON = 96
SW_XY_OVERLAP_HORIZON = 0.50

## Sliding Windows Parameters - Foreground ##
SW_YSTART_FOREGROUND = 345
SW_YSTOP_FOREGROUND = 665
SW_XY_WINDOW_FOREGROUND = 160
SW_XY_OVERLAP_FOREGROUND = 0.50


SW_SPATIAL_FEAT_FLAG = True
SW_HOG_FEAT_FLAG = True
SW_COLOR_HIST_FEAT_FLAG = True
In [65]:
# Define a function to draw bounding boxes
def draw_boxes(img, bboxes, color=BBOX_COLOR, thick=BBOX_THICK):
    # Make a copy of the image
    imcopy = np.copy(img)
    # Iterate through the bounding boxes
    for bbox in bboxes:
        # Draw a rectangle given bbox coordinates
        cv2.rectangle(imcopy, bbox[0], bbox[1], color, thick)
    return imcopy
In [66]:
# Define a function that takes an image,
# start and stop positions in both x and y, 
# window size (x and y dimensions),  
# and overlap fraction (for both x and y)
def slide_window(img, x_start_stop=[None, None], y_start_stop=[None, None], 
                    xy_window=(64, 64), xy_overlap=(0.5, 0.5)):
    # If x and/or y start/stop positions not defined, set to image size
    if x_start_stop[0] == None:
        x_start_stop[0] = 0
    if x_start_stop[1] == None:
        x_start_stop[1] = img.shape[1]
    if y_start_stop[0] == None:
        y_start_stop[0] = 0
    if y_start_stop[1] == None:
        y_start_stop[1] = img.shape[0]
    # Compute the span of the region to be searched    
    xspan = x_start_stop[1] - x_start_stop[0]
    yspan = y_start_stop[1] - y_start_stop[0]
    # Compute the number of pixels per step in x/y
    nx_pix_per_step = np.int(xy_window[0]*(1 - xy_overlap[0]))
    ny_pix_per_step = np.int(xy_window[1]*(1 - xy_overlap[1]))
    # Compute the number of windows in x/y
    nx_buffer = np.int(xy_window[0]*(xy_overlap[0]))
    ny_buffer = np.int(xy_window[1]*(xy_overlap[1]))
    nx_windows = np.int((xspan-nx_buffer)/nx_pix_per_step) 
    ny_windows = np.int((yspan-ny_buffer)/ny_pix_per_step) 
    # Initialize a list to append window positions to
    window_list = []
    # Loop through finding x and y window positions
    for ys in range(ny_windows):
        for xs in range(nx_windows):
            # Calculate window position
            startx = xs*nx_pix_per_step + x_start_stop[0]
            endx = startx + xy_window[0]
            starty = ys*ny_pix_per_step + y_start_stop[0]
            endy = starty + xy_window[1]
            # Append window position to list
            window_list.append(((startx, starty), (endx, endy)))
    return window_list
In [67]:
#Define a function for plotting multipole images
def visualize(fig, rows, cols, imgs, titles):
    for i, img in enumerate(imgs):
        plt.subplot(rows, cols, i+1)
        plt.title(i+1)
        img_dims = len(img.shape)
        if img_dims < 3:
            plt.imshow(img, cmap='hot')
            plt.title(titles[i])
        else:
            plt.imshow(img)
            plt.title(titles[i])
In [68]:
# Define a function to extract features from a single image window
# This function is very similar to extract_features()
# just for a single image rather than list of images
def single_img_features(img, color_space=COLOR_SPACE, spatial_size=(32, 32),
                        hist_bins=32, orient=9, 
                        pix_per_cell=8, cell_per_block=2, hog_channel=0,
                        spatial_feat=True, hist_feat=True, hog_feat=True):    
    #1) Define an empty list to receive features
    img_features = []
    #2) Apply color conversion if other than 'RGB'
    if color_space != 'RGB':
        if color_space == 'HSV':
            feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
        elif color_space == 'LUV':
            feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2LUV)
        elif color_space == 'HLS':
            feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
        elif color_space == 'YUV':
            feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
        elif color_space == 'YCrCb':
            feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2YCrCb)
    else: feature_image = np.copy(img)      
    #3) Compute spatial features if flag is set
    if spatial_feat == True:
        spatial_features = bin_spatial(feature_image, size=spatial_size)
        #4) Append features to list
        img_features.append(spatial_features)
    #5) Compute histogram features if flag is set
    if hist_feat == True:
        hist_features = color_hist(feature_image, nbins=hist_bins)
        #6) Append features to list
        img_features.append(hist_features)
    #7) Compute HOG features if flag is set
    if hog_feat == True:
        if hog_channel == 'ALL':
            hog_features = []
            for channel in range(feature_image.shape[2]):
                hog_features.extend(get_hog_features(feature_image[:,:,channel], 
                                    orient, pix_per_cell, cell_per_block, 
                                    vis=False, feature_vec=True))      
        else:
            hog_features = get_hog_features(feature_image[:,:,hog_channel], orient, 
                        pix_per_cell, cell_per_block, vis=False, feature_vec=True)
        #8) Append features to list
        img_features.append(hog_features)

    #9) Return concatenated array of features
    return np.concatenate(img_features)
In [69]:
# Define a function you will pass an image 
# and the list of windows to be searched (output of slide_windows())
def search_windows(img, windows, svc=SVC, X_scaler=X_SCALER, color_space=COLOR_SPACE, 
                    spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS, 
                    orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK, 
                    hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG, 
                    hist_feat=SW_COLOR_HIST_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG):
    

    #1) Create an empty list to receive positive detection windows
    on_windows = []
    #2) Iterate over all windows in the list
    for window in windows:
        #3) Extract the test window from original image
        test_img = cv2.resize(img[window[0][1]:window[1][1], window[0][0]:window[1][0]], (DEFAULT_LENGTH, DEFAULT_WIDTH))      
        #4) Extract features for that window using single_img_features()
        features = single_img_features(test_img, color_space=color_space, 
                            spatial_size=spatial_size, hist_bins=hist_bins, 
                            orient=orient, pix_per_cell=pix_per_cell, 
                            cell_per_block=cell_per_block, 
                            hog_channel=hog_channel, spatial_feat=spatial_feat, 
                            hist_feat=hist_feat, hog_feat=hog_feat)
        #5) Scale extracted features to be fed to classifier
        test_features = X_scaler.transform(np.array(features).reshape(1, -1))
        #6) Predict using your classifier
        prediction = svc.predict(test_features)
        #7) If positive (prediction == 1) then save the window
        if prediction == 1: # Car detected
            on_windows.append(window)
    #8) Return windows for positive detections
    return on_windows
    
In [70]:
# Try Scaling Windows on Test Images
image_paths = glob.glob(testset_path, recursive=True)
print('Found',len(image_paths),'images in directory:', testset_path)
Found 6 images in directory: data/test_images/*.jpg
In [71]:
# Define a single function that can extract features using hog sub-sampling and make predictions

images = []
titles = []
y_start_stop_horizon = [SW_YSTART_HORIZON, SW_YSTOP_HORIZON] # Min/Max in y to search in slide_window()
y_start_stop_foreground = [SW_YSTART_FOREGROUND, SW_YSTOP_FOREGROUND] # Min/Max in y to search in slide_window()

for img_path in image_paths:
    t1 = time.time()
    img = mpimg.imread(img_path)
    draw_img = np.copy(img)
    img = normalize_pixels(img).astype(np.float32)
    windows = []
    hot_windows = []
    
    
    # Horizon Check
    windows_1 = slide_window(img, x_start_stop=[None,None], y_start_stop=y_start_stop_horizon, 
                           xy_window=(SW_XY_WINDOW_HORIZON,SW_XY_WINDOW_HORIZON), 
                           xy_overlap=(SW_XY_OVERLAP_HORIZON,SW_XY_OVERLAP_HORIZON))

    hot_windows_1 = search_windows(img, windows_1, SVC, X_SCALER, color_space=COLOR_SPACE,
                                 spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS, orient=HOG_ORIENTATIONS,
                                 pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
                                 hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG,
                                 hist_feat=SW_COLOR_HIST_FEAT_FLAG)

    print('BBoxes Found - Horizon:', len(hot_windows_1))
    windows.extend(windows_1)
    hot_windows.extend(hot_windows_1)
    
    # Foreground Check
    windows_2 = slide_window(img, x_start_stop=[None,None], y_start_stop=y_start_stop_foreground, 
                           xy_window=(SW_XY_WINDOW_FOREGROUND,SW_XY_WINDOW_FOREGROUND), 
                           xy_overlap=(SW_XY_OVERLAP_FOREGROUND,SW_XY_OVERLAP_FOREGROUND))

    hot_windows_2 = search_windows(img, windows_2, SVC, X_SCALER, color_space=COLOR_SPACE,
                                 spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS, orient=HOG_ORIENTATIONS,
                                 pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
                                 hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG,
                                 hist_feat=SW_COLOR_HIST_FEAT_FLAG)


    print('BBoxes Found - Foreground:', len(hot_windows_2))
    windows.extend(windows_2)
    hot_windows.extend(hot_windows_2)

    window_img = draw_boxes(draw_img, hot_windows, color=BBOX_COLOR, thick=BBOX_THICK)
    images.append(window_img)
    titles.append('')
    
    print(time.time()-t1, 'seconds to process one image search', len(windows), 'windows')
    
fig = plt.figure(figsize=(12,18), dpi=300)
visualize(fig, 5, 2, images, titles)
BBoxes Found - Horizon: 4
BBoxes Found - Foreground: 1
0.8273510932922363 seconds to process one image search 95 windows
BBoxes Found - Horizon: 0
BBoxes Found - Foreground: 0
0.6296329498291016 seconds to process one image search 95 windows
BBoxes Found - Horizon: 1
BBoxes Found - Foreground: 0
0.9677538871765137 seconds to process one image search 95 windows
BBoxes Found - Horizon: 3
BBoxes Found - Foreground: 1
0.8046660423278809 seconds to process one image search 95 windows
BBoxes Found - Horizon: 4
BBoxes Found - Foreground: 1
0.6795129776000977 seconds to process one image search 95 windows
BBoxes Found - Horizon: 4
BBoxes Found - Foreground: 0
0.7243568897247314 seconds to process one image search 95 windows

We want a more efficient way to detect vehicles. This approach will allow for only a single call to get HOG features. The pipeline will then find a sub sample

In [72]:
def convert_color(img, conv=SW_CONVERT_COLOR):
    if conv == 'RGB2YCrCb':
        return cv2.cvtColor(img, cv2.COLOR_RGB2YCrCb)
    if conv == 'BGR2YCrCb':
        return cv2.cvtColor(img, cv2.COLOR_BGR2YCrCb)
    if conv == 'RGB2LUV':
        return cv2.cvtColor(img, cv2.COLOR_RGB2LUV)
In [87]:
# Define a single function that can extract features using hog sub-sampling and make predictions
out_images = []
out_maps = []
out_titles = []
out_boxes = []

## Sliding Window Parameters - HOG Sub-Sampling ##
SW_CONVERT_COLOR = 'RGB2YCrCb'
SW_YSTART = 400
SW_YSTOP = 665
SW_SCALE = 1.5

ystart = SW_YSTART
ystop = SW_YSTOP
scale = SW_SCALE
spatial_size=BIN_SPATIAL_SIZE
hist_bins=HIST_NBINS
orient=HOG_ORIENTATIONS
pix_per_cell=HOG_PIXELS_PER_CELL
cell_per_block=HOG_CELLS_PER_BLOCK
hog_channel=HOG_CHANNEL
spatial_feat=SW_SPATIAL_FEAT_FLAG
hog_feat=SW_HOG_FEAT_FLAG
hist_feat=SW_COLOR_HIST_FEAT_FLAG


#Iterate over the test images
for img_path in image_paths:
    img_boxes = []
    t1 = time.time()
    count = 0
    img = mpimg.imread(img_path)
    draw_img = np.copy(img)
    
    #Make a heatmap of zeros
    heatmap = np.zeros_like(img[:,:,0])
    img = normalize_pixels(img).astype(np.float32)
    img_to_search = img[ystart:ystop,:,:]
    ctrans_tosearch = convert_color(img_to_search, conv=SW_CONVERT_COLOR)
    if scale != 1:
        imshape = ctrans_tosearch.shape
        ctrans_tosearch = cv2.resize(ctrans_tosearch, (np.int(imshape[1]/scale), np.int(imshape[0]/scale)))

    ch1 = ctrans_tosearch[:,:,0]
    ch2 = ctrans_tosearch[:,:,1]
    ch3 = ctrans_tosearch[:,:,2]
    
    # Define blocks and steps as above
    nxblocks = (ch1.shape[1] // pix_per_cell)-1
    nyblocks = (ch1.shape[0] // pix_per_cell)-1 
    nfeat_per_block = orient*cell_per_block**2
    window = HOG_PIXELS_PER_CELL*HOG_PIXELS_PER_CELL # 8 cells and 8 pix per cell
    nblocks_per_window = (window // pix_per_cell)-1  # The // division is used for integers (for indices)
    cells_per_step = 2  # Instead of overlap, define how many cells to step
    nxsteps = (nxblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
    nysteps = (nyblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
    
    # Compute individual channel HOG features for the entire image
    hog1 = get_hog_features(ch1, orient, pix_per_cell, cell_per_block, feature_vec=False)
    hog2 = get_hog_features(ch2, orient, pix_per_cell, cell_per_block, feature_vec=False)
    hog3 = get_hog_features(ch3, orient, pix_per_cell, cell_per_block, feature_vec=False)


    for xb in range(nxsteps):
        for yb in range(nysteps):
            count += 1
            ypos = yb*cells_per_step
            xpos = xb*cells_per_step

            # Extract HOG for this particular patch
            hog_feat1 = hog1[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_feat2 = hog2[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_feat3 = hog3[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_features = np.hstack((hog_feat1, hog_feat2, hog_feat3))
            
            xleft = xpos*pix_per_cell
            ytop = ypos*pix_per_cell
            
            # Extract the image patch
            subimg = cv2.resize(ctrans_tosearch[ytop:ytop+window, xleft:xleft+window], (DEFAULT_LENGTH, DEFAULT_WIDTH))

            # Get color features
            spatial_features = bin_spatial(subimg, size=BIN_SPATIAL_SIZE)
            hist_features = color_hist(subimg, nbins=HIST_NBINS)
            
            
            test_feats = np.hstack((spatial_features, hist_features, hog_features))
            # Scale features and make a prediction
            test_features = X_scaler.transform(test_feats.reshape(1, -1))
            test_prediction = svc.predict(test_features)

            if test_prediction == 1:
                xbox_left = np.int(xleft*scale)
                ytop_draw = np.int(ytop*scale)
                win_draw = np.int(window*scale)
                cv2.rectangle(draw_img,(xbox_left, ytop_draw+ystart),
                              (xbox_left+win_draw, ytop_draw+win_draw+ystart),BBOX_COLOR,6)
                img_boxes.append(((xbox_left, ytop_draw+ystart),
                              (xbox_left+win_draw,ytop_draw+win_draw+ystart)))

                heatmap[ytop_draw+ystart:ytop_draw+win_draw+ystart, xbox_left:xbox_left+win_draw] +=1
        print(time.time()-t1, 'seconds to run, total windows = ', count)

    out_images.append(draw_img)

    out_titles.append(os.path.split(img_path)[-1])
    out_titles.append(os.path.split(img_path)[-1])
    out_images.append(heatmap)
    out_maps.append(heatmap)
    out_boxes.append(img_boxes)
    
fig = plt.figure(figsize=(12,24))
visualize(fig, 8, 2, out_images, out_titles)
0.5941650867462158 seconds to run, total windows =  7
0.6092400550842285 seconds to run, total windows =  14
0.6209070682525635 seconds to run, total windows =  21
0.6409010887145996 seconds to run, total windows =  28
0.6494529247283936 seconds to run, total windows =  35
0.6593749523162842 seconds to run, total windows =  42
0.6656661033630371 seconds to run, total windows =  49
0.6714611053466797 seconds to run, total windows =  56
0.6802899837493896 seconds to run, total windows =  63
0.7006261348724365 seconds to run, total windows =  70
0.7072141170501709 seconds to run, total windows =  77
0.7157230377197266 seconds to run, total windows =  84
0.734807014465332 seconds to run, total windows =  91
0.7426929473876953 seconds to run, total windows =  98
0.7486469745635986 seconds to run, total windows =  105
0.7564680576324463 seconds to run, total windows =  112
0.762382984161377 seconds to run, total windows =  119
0.7683091163635254 seconds to run, total windows =  126
0.7741811275482178 seconds to run, total windows =  133
0.7799999713897705 seconds to run, total windows =  140
0.7867679595947266 seconds to run, total windows =  147
0.7925219535827637 seconds to run, total windows =  154
0.7995569705963135 seconds to run, total windows =  161
0.8075530529022217 seconds to run, total windows =  168
0.8143079280853271 seconds to run, total windows =  175
0.8208341598510742 seconds to run, total windows =  182
0.8269240856170654 seconds to run, total windows =  189
0.8334591388702393 seconds to run, total windows =  196
0.840487003326416 seconds to run, total windows =  203
0.8484251499176025 seconds to run, total windows =  210
0.8588860034942627 seconds to run, total windows =  217
0.8686749935150146 seconds to run, total windows =  224
0.8898000717163086 seconds to run, total windows =  231
0.9038429260253906 seconds to run, total windows =  238
0.9116201400756836 seconds to run, total windows =  245
0.9187560081481934 seconds to run, total windows =  252
0.9306631088256836 seconds to run, total windows =  259
0.9390659332275391 seconds to run, total windows =  266
0.9446780681610107 seconds to run, total windows =  273
0.9503331184387207 seconds to run, total windows =  280
0.9577770233154297 seconds to run, total windows =  287
0.9639010429382324 seconds to run, total windows =  294
0.9704220294952393 seconds to run, total windows =  301
0.9768891334533691 seconds to run, total windows =  308
0.9831709861755371 seconds to run, total windows =  315
0.9992260932922363 seconds to run, total windows =  322
1.0069050788879395 seconds to run, total windows =  329
1.0140020847320557 seconds to run, total windows =  336
1.02030611038208 seconds to run, total windows =  343
0.3296849727630615 seconds to run, total windows =  7
0.33577394485473633 seconds to run, total windows =  14
0.34278082847595215 seconds to run, total windows =  21
0.3500359058380127 seconds to run, total windows =  28
0.3557560443878174 seconds to run, total windows =  35
0.36159205436706543 seconds to run, total windows =  42
0.3672499656677246 seconds to run, total windows =  49
0.37325596809387207 seconds to run, total windows =  56
0.37931299209594727 seconds to run, total windows =  63
0.38543200492858887 seconds to run, total windows =  70
0.39218783378601074 seconds to run, total windows =  77
0.3988020420074463 seconds to run, total windows =  84
0.4047060012817383 seconds to run, total windows =  91
0.4109318256378174 seconds to run, total windows =  98
0.4179389476776123 seconds to run, total windows =  105
0.42414093017578125 seconds to run, total windows =  112
0.42989301681518555 seconds to run, total windows =  119
0.43567991256713867 seconds to run, total windows =  126
0.44197893142700195 seconds to run, total windows =  133
0.4478728771209717 seconds to run, total windows =  140
0.45405101776123047 seconds to run, total windows =  147
0.4596560001373291 seconds to run, total windows =  154
0.46580004692077637 seconds to run, total windows =  161
0.47170495986938477 seconds to run, total windows =  168
0.4780998229980469 seconds to run, total windows =  175
0.48372888565063477 seconds to run, total windows =  182
0.4897949695587158 seconds to run, total windows =  189
0.49590492248535156 seconds to run, total windows =  196
0.5016570091247559 seconds to run, total windows =  203
0.5073409080505371 seconds to run, total windows =  210
0.5135760307312012 seconds to run, total windows =  217
0.5206589698791504 seconds to run, total windows =  224
0.5271780490875244 seconds to run, total windows =  231
0.5341808795928955 seconds to run, total windows =  238
0.5414509773254395 seconds to run, total windows =  245
0.5480320453643799 seconds to run, total windows =  252
0.5549800395965576 seconds to run, total windows =  259
0.5614359378814697 seconds to run, total windows =  266
0.5684258937835693 seconds to run, total windows =  273
0.5755748748779297 seconds to run, total windows =  280
0.5934920310974121 seconds to run, total windows =  287
0.6090459823608398 seconds to run, total windows =  294
0.6164309978485107 seconds to run, total windows =  301
0.629910945892334 seconds to run, total windows =  308
0.636091947555542 seconds to run, total windows =  315
0.6417558193206787 seconds to run, total windows =  322
0.6484308242797852 seconds to run, total windows =  329
0.655052900314331 seconds to run, total windows =  336
0.6623589992523193 seconds to run, total windows =  343
0.25664591789245605 seconds to run, total windows =  7
0.2742729187011719 seconds to run, total windows =  14
0.2808988094329834 seconds to run, total windows =  21
0.28705501556396484 seconds to run, total windows =  28
0.29388880729675293 seconds to run, total windows =  35
0.30064988136291504 seconds to run, total windows =  42
0.3078029155731201 seconds to run, total windows =  49
0.31428098678588867 seconds to run, total windows =  56
0.32024192810058594 seconds to run, total windows =  63
0.3283729553222656 seconds to run, total windows =  70
0.3359718322753906 seconds to run, total windows =  77
0.34326982498168945 seconds to run, total windows =  84
0.34919095039367676 seconds to run, total windows =  91
0.35534095764160156 seconds to run, total windows =  98
0.36150288581848145 seconds to run, total windows =  105
0.36736202239990234 seconds to run, total windows =  112
0.3729069232940674 seconds to run, total windows =  119
0.37990593910217285 seconds to run, total windows =  126
0.3866310119628906 seconds to run, total windows =  133
0.39280080795288086 seconds to run, total windows =  140
0.3984808921813965 seconds to run, total windows =  147
0.4054238796234131 seconds to run, total windows =  154
0.41123199462890625 seconds to run, total windows =  161
0.41689586639404297 seconds to run, total windows =  168
0.422576904296875 seconds to run, total windows =  175
0.4318428039550781 seconds to run, total windows =  182
0.4404928684234619 seconds to run, total windows =  189
0.46027493476867676 seconds to run, total windows =  196
0.4675579071044922 seconds to run, total windows =  203
0.484332799911499 seconds to run, total windows =  210
0.4921150207519531 seconds to run, total windows =  217
0.5005228519439697 seconds to run, total windows =  224
0.5143318176269531 seconds to run, total windows =  231
0.5246078968048096 seconds to run, total windows =  238
0.5315628051757812 seconds to run, total windows =  245
0.539099931716919 seconds to run, total windows =  252
0.5467178821563721 seconds to run, total windows =  259
0.556725025177002 seconds to run, total windows =  266
0.56430983543396 seconds to run, total windows =  273
0.5821268558502197 seconds to run, total windows =  280
0.592789888381958 seconds to run, total windows =  287
0.6008749008178711 seconds to run, total windows =  294
0.6093709468841553 seconds to run, total windows =  301
0.6192059516906738 seconds to run, total windows =  308
0.630422830581665 seconds to run, total windows =  315
0.6488828659057617 seconds to run, total windows =  322
0.6618859767913818 seconds to run, total windows =  329
0.6822190284729004 seconds to run, total windows =  336
0.7328019142150879 seconds to run, total windows =  343
0.365123987197876 seconds to run, total windows =  7
0.3718221187591553 seconds to run, total windows =  14
0.37746310234069824 seconds to run, total windows =  21
0.3833191394805908 seconds to run, total windows =  28
0.3893270492553711 seconds to run, total windows =  35
0.3952469825744629 seconds to run, total windows =  42
0.4012441635131836 seconds to run, total windows =  49
0.4072740077972412 seconds to run, total windows =  56
0.4132421016693115 seconds to run, total windows =  63
0.4190061092376709 seconds to run, total windows =  70
0.42516303062438965 seconds to run, total windows =  77
0.43215513229370117 seconds to run, total windows =  84
0.438291072845459 seconds to run, total windows =  91
0.44510698318481445 seconds to run, total windows =  98
0.4516489505767822 seconds to run, total windows =  105
0.45751500129699707 seconds to run, total windows =  112
0.46425509452819824 seconds to run, total windows =  119
0.47016096115112305 seconds to run, total windows =  126
0.475844144821167 seconds to run, total windows =  133
0.4830200672149658 seconds to run, total windows =  140
0.48923206329345703 seconds to run, total windows =  147
0.49553513526916504 seconds to run, total windows =  154
0.501255989074707 seconds to run, total windows =  161
0.5073051452636719 seconds to run, total windows =  168
0.5178720951080322 seconds to run, total windows =  175
0.5365500450134277 seconds to run, total windows =  182
0.5452151298522949 seconds to run, total windows =  189
0.5542910099029541 seconds to run, total windows =  196
0.5769340991973877 seconds to run, total windows =  203
0.5853409767150879 seconds to run, total windows =  210
0.602288007736206 seconds to run, total windows =  217
0.6088640689849854 seconds to run, total windows =  224
0.615645170211792 seconds to run, total windows =  231
0.6272449493408203 seconds to run, total windows =  238
0.6352500915527344 seconds to run, total windows =  245
0.6467790603637695 seconds to run, total windows =  252
0.6554510593414307 seconds to run, total windows =  259
0.6722500324249268 seconds to run, total windows =  266
0.6875030994415283 seconds to run, total windows =  273
0.6953020095825195 seconds to run, total windows =  280
0.704456090927124 seconds to run, total windows =  287
0.7176151275634766 seconds to run, total windows =  294
0.7237710952758789 seconds to run, total windows =  301
0.7386820316314697 seconds to run, total windows =  308
0.7466330528259277 seconds to run, total windows =  315
0.7525441646575928 seconds to run, total windows =  322
0.7587471008300781 seconds to run, total windows =  329
0.7647891044616699 seconds to run, total windows =  336
0.7712740898132324 seconds to run, total windows =  343
0.32031893730163574 seconds to run, total windows =  7
0.3266561031341553 seconds to run, total windows =  14
0.3327059745788574 seconds to run, total windows =  21
0.3383760452270508 seconds to run, total windows =  28
0.34427404403686523 seconds to run, total windows =  35
0.3508181571960449 seconds to run, total windows =  42
0.3569600582122803 seconds to run, total windows =  49
0.3628239631652832 seconds to run, total windows =  56
0.3685619831085205 seconds to run, total windows =  63
0.3743710517883301 seconds to run, total windows =  70
0.38056492805480957 seconds to run, total windows =  77
0.38620805740356445 seconds to run, total windows =  84
0.3922851085662842 seconds to run, total windows =  91
0.39966607093811035 seconds to run, total windows =  98
0.4059121608734131 seconds to run, total windows =  105
0.41211509704589844 seconds to run, total windows =  112
0.4177520275115967 seconds to run, total windows =  119
0.42348504066467285 seconds to run, total windows =  126
0.42931699752807617 seconds to run, total windows =  133
0.43494200706481934 seconds to run, total windows =  140
0.4408690929412842 seconds to run, total windows =  147
0.4476909637451172 seconds to run, total windows =  154
0.45372605323791504 seconds to run, total windows =  161
0.45946693420410156 seconds to run, total windows =  168
0.4655759334564209 seconds to run, total windows =  175
0.47144198417663574 seconds to run, total windows =  182
0.47703003883361816 seconds to run, total windows =  189
0.48454809188842773 seconds to run, total windows =  196
0.4907200336456299 seconds to run, total windows =  203
0.4967050552368164 seconds to run, total windows =  210
0.5029420852661133 seconds to run, total windows =  217
0.5107030868530273 seconds to run, total windows =  224
0.5175340175628662 seconds to run, total windows =  231
0.5275521278381348 seconds to run, total windows =  238
0.5347750186920166 seconds to run, total windows =  245
0.5407111644744873 seconds to run, total windows =  252
0.5472960472106934 seconds to run, total windows =  259
0.5548141002655029 seconds to run, total windows =  266
0.5606191158294678 seconds to run, total windows =  273
0.5662930011749268 seconds to run, total windows =  280
0.5725359916687012 seconds to run, total windows =  287
0.5781290531158447 seconds to run, total windows =  294
0.5841670036315918 seconds to run, total windows =  301
0.5898470878601074 seconds to run, total windows =  308
0.5960249900817871 seconds to run, total windows =  315
0.6019561290740967 seconds to run, total windows =  322
0.6079239845275879 seconds to run, total windows =  329
0.6139750480651855 seconds to run, total windows =  336
0.6195700168609619 seconds to run, total windows =  343
0.2512640953063965 seconds to run, total windows =  7
0.25776195526123047 seconds to run, total windows =  14
0.2645890712738037 seconds to run, total windows =  21
0.27166199684143066 seconds to run, total windows =  28
0.2786681652069092 seconds to run, total windows =  35
0.2856719493865967 seconds to run, total windows =  42
0.29262495040893555 seconds to run, total windows =  49
0.2984189987182617 seconds to run, total windows =  56
0.30410003662109375 seconds to run, total windows =  63
0.30966997146606445 seconds to run, total windows =  70
0.3154301643371582 seconds to run, total windows =  77
0.3216731548309326 seconds to run, total windows =  84
0.3273930549621582 seconds to run, total windows =  91
0.33323216438293457 seconds to run, total windows =  98
0.33907294273376465 seconds to run, total windows =  105
0.34500813484191895 seconds to run, total windows =  112
0.3508720397949219 seconds to run, total windows =  119
0.35647106170654297 seconds to run, total windows =  126
0.36250805854797363 seconds to run, total windows =  133
0.3682999610900879 seconds to run, total windows =  140
0.374269962310791 seconds to run, total windows =  147
0.3805661201477051 seconds to run, total windows =  154
0.3866879940032959 seconds to run, total windows =  161
0.39394211769104004 seconds to run, total windows =  168
0.4004950523376465 seconds to run, total windows =  175
0.4060380458831787 seconds to run, total windows =  182
0.4134070873260498 seconds to run, total windows =  189
0.4207730293273926 seconds to run, total windows =  196
0.42667317390441895 seconds to run, total windows =  203
0.43289899826049805 seconds to run, total windows =  210
0.43904805183410645 seconds to run, total windows =  217
0.4447319507598877 seconds to run, total windows =  224
0.45154500007629395 seconds to run, total windows =  231
0.4584381580352783 seconds to run, total windows =  238
0.46450209617614746 seconds to run, total windows =  245
0.47034502029418945 seconds to run, total windows =  252
0.47794413566589355 seconds to run, total windows =  259
0.4847540855407715 seconds to run, total windows =  266
0.4917421340942383 seconds to run, total windows =  273
0.4989809989929199 seconds to run, total windows =  280
0.5057501792907715 seconds to run, total windows =  287
0.5121469497680664 seconds to run, total windows =  294
0.518449068069458 seconds to run, total windows =  301
0.5251779556274414 seconds to run, total windows =  308
0.532081127166748 seconds to run, total windows =  315
0.5387911796569824 seconds to run, total windows =  322
0.5462930202484131 seconds to run, total windows =  329
0.5533721446990967 seconds to run, total windows =  336
0.5594179630279541 seconds to run, total windows =  343
In [90]:
## Sliding Window Parameters - HOG Sub-Sampling ##
SW_CONVERT_COLOR = 'RGB2YCrCb'
SW_YSTART = 400
SW_YSTOP = 665
SW_SCALE = 1.5

# Define a single function that can extract features using hog sub-sampling and make predictions
def find_cars(img, ystart=SW_YSTART, ystop=SW_YSTOP, scale=SW_SCALE, svc=SVC, X_scaler=X_SCALER,
              orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
              spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS):

    spatial_feat=SW_SPATIAL_FEAT_FLAG
    hog_feat=SW_HOG_FEAT_FLAG
    hist_feat=SW_COLOR_HIST_FEAT_FLAG

    draw_img = np.copy(img)

    #Make a heatmap of zeros
    heatmap = np.zeros_like(img[:,:,0])
    img = normalize_pixels(img).astype(np.float32)
    img_to_search = img[ystart:ystop,:,:]
    ctrans_tosearch = convert_color(img_to_search, conv=SW_CONVERT_COLOR)
    if scale != 1:
        imshape = ctrans_tosearch.shape
        ctrans_tosearch = cv2.resize(ctrans_tosearch, (np.int(imshape[1]/scale), np.int(imshape[0]/scale)))

    ch1 = ctrans_tosearch[:,:,0]
    ch2 = ctrans_tosearch[:,:,1]
    ch3 = ctrans_tosearch[:,:,2]

    # Define blocks and steps as above
    nxblocks = (ch1.shape[1] // pix_per_cell)-1
    nyblocks = (ch1.shape[0] // pix_per_cell)-1 
    nfeat_per_block = orient*cell_per_block**2
    window = HOG_PIXELS_PER_CELL*HOG_PIXELS_PER_CELL # 8 cells and 8 pix per cell
    nblocks_per_window = (window // pix_per_cell)-1  # The // division is used for integers (for indices)
    cells_per_step = 2  # Instead of overlap, define how many cells to step
    nxsteps = (nxblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
    nysteps = (nyblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)

    # Compute individual channel HOG features for the entire image
    hog1 = get_hog_features(ch1, orient, pix_per_cell, cell_per_block, feature_vec=False)
    hog2 = get_hog_features(ch2, orient, pix_per_cell, cell_per_block, feature_vec=False)
    hog3 = get_hog_features(ch3, orient, pix_per_cell, cell_per_block, feature_vec=False)


    for xb in range(nxsteps):
        for yb in range(nysteps):
            ypos = yb*cells_per_step
            xpos = xb*cells_per_step

            # Extract HOG for this particular patch
            hog_feat1 = hog1[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_feat2 = hog2[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_feat3 = hog3[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel() 
            hog_features = np.hstack((hog_feat1, hog_feat2, hog_feat3))

            xleft = xpos*pix_per_cell
            ytop = ypos*pix_per_cell

            # Extract the image patch
            subimg = cv2.resize(ctrans_tosearch[ytop:ytop+window, xleft:xleft+window], (DEFAULT_LENGTH, DEFAULT_WIDTH))

            # Get color features
            spatial_features = bin_spatial(subimg, size=BIN_SPATIAL_SIZE)
            hist_features = color_hist(subimg, nbins=HIST_NBINS)

            test_feats = np.hstack((spatial_features, hist_features, hog_features))
            # Scale features and make a prediction
            test_features = X_scaler.transform(test_feats.reshape(1, -1))
            test_prediction = svc.predict(test_features)

            if test_prediction == 1:
                xbox_left = np.int(xleft*scale)
                ytop_draw = np.int(ytop*scale)
                win_draw = np.int(window*scale)
                cv2.rectangle(draw_img,(xbox_left, ytop_draw+ystart),
                              (xbox_left+win_draw, ytop_draw+win_draw+ystart),BBOX_COLOR,6)
                heatmap[ytop_draw+ystart:ytop_draw+win_draw+ystart, xbox_left:xbox_left+win_draw] +=1
    return draw_img, heatmap
In [96]:
from scipy.ndimage.measurements import label

def apply_threshold(heatmap, threshold):
    # Zero out pixels below the threshold
    heatmap[heatmap <= threshold] = 0
    return heatmap

def draw_labeled_bboxes(img, labels):
    # Iterate through all detected cars
    for car_number in range(1, labels[1] + 1):
        #Find pixels with each car_number label value
        nonzero = (labels[0] == car_number).nonzero()
        # Identify x and y values of those pixels
        nonzeroy = np.array(nonzero[0])
        nonzerox = np.array(nonzero[1])
        #Define a bounding box based on min/max x and y
        bbox = ((np.min(nonzerox), np.min(nonzeroy)), (np.max(nonzerox), np.max(nonzeroy)))
        #Draw the box on the image
        cv2.rectangle(img, bbox[0], bbox[1], BBOX_COLOR, BBOX_THICK)
    # Return the image
    return img
In [101]:
out_images = []
out_maps = []
out_titles = []
out_boxes = []

## Sliding Window Parameters - HOG Sub-Sampling ##
SW_CONVERT_COLOR = 'RGB2YCrCb'
SW_YSTART = 400
SW_YSTOP = 665
SW_SCALE = 1.5


#Iterate over the test images
for img_path in image_paths:
    img = mpimg.imread(img_path)
    out_img, heatmap = find_cars(img, scale=SW_SCALE)
    labels = label(apply_threshold(heatmap, 2))
    # Draw bounding boxes on a copy of the image       
    draw_img = draw_labeled_bboxes(np.copy(img), labels)
    out_images.append(draw_img)
    out_images.append(heatmap)
    out_titles.append(os.path.split(img_path)[-1])
    out_titles.append(os.path.split(img_path)[-1])

fig = plt.figure(figsize=(12,24))
visualize(fig, 8, 2, out_images, out_titles)
In [102]:
def process_image(img):
    out_img, heatmap = find_cars(img, scale=SW_SCALE)
    labels = label(apply_threshold(heatmap, 2))
    # Draw bounding boxes on a copy of the image       
    draw_img = draw_labeled_bboxes(np.copy(img), labels)
    return draw_img
In [105]:
#Import packages to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML

test_ouput = 'test_output.mp4'
TEST_VIDEO = 'test_video.mp4'
PROJECT_VIDEO = 'project_video.mp4'
VIDEO_FILE_PATH = os.path.join(WORKING_DIRECTORY, TEST_VIDEO)
print(VIDEO_FILE_PATH)
clip = VideoFileClip(VIDEO_FILE_PATH)
test_clip = clip.fl_image(process_image)
#&time
test_clip.write_videofile(test_ouput, audio=False)
data/test_video.mp4
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-105-7b6ce777fe94> in <module>()
      8 VIDEO_FILE_PATH = os.path.join(WORKING_DIRECTORY, TEST_VIDEO)
      9 print(VIDEO_FILE_PATH)
---> 10 clip = VideoFileClip(VIDEO_FILE_PATH)
     11 test_clip = clip.fl_image(process_image)
     12 #&time

/Users/deanmwebb/anaconda/envs/sdc_dev/lib/python3.5/site-packages/moviepy/video/io/VideoFileClip.py in __init__(self, filename, has_mask, audio, audio_buffersize, audio_fps, audio_nbytes, verbose)
     80         # Make a reader for the audio, if any.
     81         if audio and self.reader.infos['audio_found']:
---> 82 
     83             self.audio = AudioFileClip(filename,
     84                                        buffersize= audio_buffersize,

/Users/deanmwebb/anaconda/envs/sdc_dev/lib/python3.5/site-packages/moviepy/audio/io/AudioFileClip.py in __init__(self, filename, buffersize, nbytes, fps)
     61         self.filename = filename
     62         reader = FFMPEG_AudioReader(filename,fps=fps,nbytes=nbytes,
---> 63                                          buffersize=buffersize)
     64 
     65         self.reader = reader

/Users/deanmwebb/anaconda/envs/sdc_dev/lib/python3.5/site-packages/moviepy/audio/io/readers.py in __init__(self, filename, buffersize, print_infos, fps, nbytes, nchannels)
     68         self.buffer_startframe = 1
     69         self.initialize()
---> 70         self.buffer_around(1)
     71 
     72 

/Users/deanmwebb/anaconda/envs/sdc_dev/lib/python3.5/site-packages/moviepy/audio/io/readers.py in buffer_around(self, framenumber)
    232         else:
    233             self.seek(new_bufferstart)
--> 234             self.buffer =  self.read_chunk(self.buffersize)
    235 
    236         self.buffer_startframe = new_bufferstart

/Users/deanmwebb/anaconda/envs/sdc_dev/lib/python3.5/site-packages/moviepy/audio/io/readers.py in read_chunk(self, chunksize)
    121         result = (1.0*result / 2**(8*self.nbytes-1)).\
    122                                  reshape((int(len(result)/self.nchannels),
--> 123                                           self.nchannels))
    124         #self.proc.stdout.flush()
    125         self.pos = self.pos+chunksize

TypeError: 'float' object cannot be interpreted as an integer
In [ ]: